1. Field of the Invention
This invention relates to computer-assisted reasoning. More specifically, the invention provides a system and method for exploring solutions to complex problems by automatically generating multiple series of experiments, by permitting users to visualize outcomes of experiments, and by allowing users to easily generate additional series of experiments using differing input dimensions or solution strategies.
2. Description of the Related Art
People increasingly use computer-based calculations to help reason about complex problems in many fields, from scientific research to engineering design to business operations and planning. Yet, despite enormous improvements in computer capabilities over past decades, people still generally use computers to support the same style of quantitative reasoning they used before computers existed.
Even so, existing computer applications often perform calculations which produce results not obvious to the computer user. Such calculations can be considered experiments. The following are examples of experiments: a simulation model which projects the future course of the economy, a spreadsheet which combines disparate accounts into a profit and loss statement, a statistical routine which finds a pattern in a large database of chemical reaction outcomes (data mining,) a search over networked databases or the World Wide Web, and computer controlled physical experiments such as combining two chemicals to measure their reactivity. In contrast, a word processing program that displays keystrokes and places them into a computer file or a numerically controlled machine are not experiments.
Some computer applications perform several iterations of an experiment. For instance, many optimization routines can be used to make repeated calls to a simulation model in order to find the inputs which give the most desirable output. To date, however, users generally use computers to conduct iterations of experiments that are simple to specify analytically, such as by an optimization routine.
Recent studies suggested that a number of important, very difficult, and previously intractable problems can be approached by conducting a series of experiments directed to the problem. Such series of experiments, or compound experiments, can be extremely difficult and time-consuming to design and build. Thus, it has been exceedingly difficult to exploit the power of compound experiments, and the benefits have been difficult to perceive. Because of the prohibitive costs, it is simply not recognized how profoundly systematic compound experiments can enhance the analysis of previously intractable problems.
For example, computer models of the future economy and climate have been developed to answer questions phrased as optimization problems, such as, xe2x80x9cwhat is the most cost-effective policy response, assuming we can accurately characterize the future behavior of the climate and economy over the next century.xe2x80x9d While these models contain much useful information, they unfortunately lack the power to make accurate predictions in the face of numerous uncertainties and overwhelming complexity. Moreover, such direct policy questions may have no resolvable answer.
On the other hand, compound experiments can be used to devise robust policies that provide reasonably good results, no matter what predictions of the future turn out to be true. This sort of approach has produced useful results in isolated cases for various business and public policy problems.
Compound experiments are a crucial tool for supporting the reasoning that can lead to understanding very complex systems. To date, however, the great cost in time and human labor to construct compound experiments has largely limited their use to only the simplest problems. Moreover, there has simply been no support for conducting large numbers of experiments automatically driven by the needs of a specific reasoning strategy.
What is needed is a general-purpose mechanism that allows the construction of a wide variety of systems that support reasoning about complex and uncertain problems using compound experiments.
One embodiment of the present invention is a system for performing compound computational experiments. The system comprises (1) an experiment having at least one input, the experiment including a process controllable by a computer, the process responsive to the at least one input to produce at least one output, the at least one output capable of being stored on a medium readable by the computer; (2) a generator capable of running on the computer, the generator automating the performance of a plurality of the experiments, the generator automatically varying the inputs for the experiments; (3) an outcomeStore storing the outputs on the medium; and (4) a display capable of running on the computer, the display presenting an interpretation of a set of the outputs extracted from the outcomeStore, the display responsive to user input to alter the interpretation, to present an interpretation of a different set of the outputs, or to cause the generator to perform a plurality of the experiments. A preferred aspect of the system further comprises a dispatch capable of running on the computer, the computer being one of a plurality of networked computers, the dispatch causing a first experiment to run on a first of the networked computers and causing a second experiment to run on a second of the networked computers. Another aspect of the system further comprises a dispatch capable of running on the computer, the dispatch causing each of the experiments to be run on a computer upon determining, by examining data in the outcomeStore, that the experiment has not previously been run. A still further aspect of the system is one wherein (1) the process includes first and second sub-models such that each of the experiments uses either the first sub-model or the second sub-model, the first sub-model responsive to first inputs, the second sub-model responsive to second inputs, the second inputs differing from the first inputs, wherein (2) the generator automatically produces sets of the first inputs and automatically varies the values of the first inputs in each of the sets of the first inputs, wherein (3) the generator automatically produces sets of the second inputs and automatically varies the values of the second inputs in each of the sets of the second inputs, and wherein (4) the generator automatically provides a set of the first inputs for each experiment using the first sub-model and automatically provides a set of the second inputs for each experiment using the second sub-model. Yet another aspect of the system further comprises (1) a second generator capable of running on the computer, the second generator automating the performance of a plurality of the experiments without changing the experiment and automatically varying the inputs for the experiments according to a sampling or search strategy different than that used by the generator; and (2) a second outcomeStore storing on the medium the outputs of the experiments automated by the second generator. Another aspect of the system is one wherein the display presents a plotted representation of two dimensions of a compound computational experiment for a first set of the outputs, the system further comprises a user interface capable of running on the computer, the user interface including a control manipulable by a user to associate a value with an unplotted dimension of the compound computational experiment, the display responsive to a manipulation of the control to (1) extract a second set of the outputs resulting from experiments that were run with a value for the unplotted dimension substantially equivalent to the value resulting from the manipulation, and to (2) present a plotted representation of the two dimensions of the compound computational experiment for the second set of the outputs. Also, another aspect of the system is one wherein the display presents a plotted representation of two dimensions of a compound computational experiment for a first set of the outputs, the system further comprises a user interface capable of running on the computer, the user interface including a control manipulable by a user to associate a value with an unplotted dimension of the compound computational experiment, the display responsive to a manipulation of the control to (1) cause the generator to perform a plurality of experiments to generate a second set of the outputs run using a value for the unplotted dimension substantially equivalent to the value resulting from the manipulation, and to (2) present a plotted representation of the two dimensions of the compound computational experiment for the second set of the outputs.
Another embodiment of the present invention is a method for performing compound computational experiments. The method comprises the steps of (1) encapsulating an experiment in an experiment object having a run method, the run method callable by other objects to run an experiment, the run method encapsulating outputs produced by an experiment in a result object; (2) encapsulating all input and output variables for an experiment in a domain object, the domain object having a read method callable by other objects to receive information about the input and output variables; and (3) encapsulating a search or sampling strategy in generator object, the generator object having a generate method callable by other objects to repetitively encapsulate a value for each input variable in a specification object and to repetitively call the run method to perform an experiment using each specification object, the generator calling the read method to acquire information about the input variables. A preferred aspect of the method comprises the further steps of (1) encapsulating experiment result data in an outcomeStore object, the outcomeStore object having an extract method callable by other objects to retrieve experiment result data; and (2) requesting a plot of experiment result data by issuing a plot command through a user interface running on a computer, the request invoking the generate method to produce experiment result data and invoking the extract method to acquire experiment result data for experiment dimensions related to the requested plot. Another aspect of method comprises the further steps of (1) plotting pixel data on a computer screen to visualize experiment result data corresponding to the experiment dimensions related to the requested plot; and (2) adjusting a control on the user interface to associate a value with an unplotted dimension of the experiment, the adjustment invoking the generate method to produce second experiment result data, invoking the extract method to acquire second experiment result data for experiment dimensions related to the requested plot, and plotting pixel data on the computer screen to visualize the second experiment result data corresponding to the experiment dimensions related to the requested plot.
Still another embodiment of the present invention is a system for performing compound computational experiments. The system comprises (1) experiment means for encapsulating and running a computational experiment; (2) domain means for encapsulating all inputs and outputs for an experiment; and (3) generator means for generating experiments by encapsulating subsets of the inputs encapsulated in the domain means and by combining the encapsulated subsets of inputs with the experiment means.